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Abstract 

A  pilot  decision  analysis  was  conducted  to  determine  an 
optimal  test  strategy  for  the  PEACEKEEPER  intercontinental 
ballistic  missile  weapon  system.  Air  Force  officers 
experienced  with  PEACEKEEPER  were  consulted  and  their  inputs 
combined  into  an  influence  diagram.  The  resulting  model  was 
solved  using  the  Performs  software  program  and  showed  that, 
while  launching  six  missiles  per  year  gives  the  largest 
expected  value,  little  additional  information  is  gained  after 
the  third  launch.  This  simple  model  indicates  that  a  new,  more 
comprehensive  decision-making  process  is  possible.  The  model 
is  flexible  and  can  be  easily  expanded  to  incorporate  any 
factors  that  a  decision  maker  might  want  to  consider  before 
expending  his  scarce  resources.  It  allows  relationships 
between  the  variables  to  be  readily  shown  and  more  fully 
understood.  This  pilot  model  is  a  starting  point  from  which 
new  insights  may  be  developed  through  the  integrated  use  of 
currently  available  weapon  system  data. 
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PEACEKEEPER  TEST  STRATEGY:  A  PILOT  DECISION  ANALYSIS 


I .  Introduction 


Background 

Intercontinental  Ballistic  Missile  (ICBM)  reliability  data 
is  used  by  the  Joint  Chiefs  of  Staff  (JCS)  in  planning  the 
Single  Integrated  Operations  Plan  (S IOP)  (6:7).  Historically, 
this  reliability  data  has  been  obtained  through  field  testing, 
using  simulated  electronic  launch-MINUTEMAN  (SELM)  exercises, 
and  flight  tests  on  the  Western  Test  Range  as  depicted  in 
Figure  1  (10).  This  was  possible  because,  until  recently, 
there  were  adequate  numbers  of  test  assets  available  to  ensure 
a  viable  test  program  for  each  of  the  different  MINUTEMAN 
configurations.  This  will  not  be  the  case  with  the  new 
PEACEKEEPER  system.  PEACEKEEPER  may  be  deployed  in  limited 
numbers,  in  more  than  one  basing  mode,  and  with  limited  test 
assets;  therefore  the  traditional  methods  for  gathering  ICBM 
reliability  data  may  no  longer  be  able  to  detect  system 
degrades  with  the  same  degree  of  confidence  as  in  the  past. 

Historically,  Strategic  Air  Command  (SAC)  planners  have 
only  used  flight  and  ground  tests  to  collect  ICBM  system 
reliability  data  (8).  The  availability  of  test  assets  allowed 
for  a  robust  flight  test  program  and  the  large  number  of 
deployed  missiles  allowed  for  11  interconnected  missiles  and 
their  two  associated  launch  control 
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Figure  1.  Test  Strategy  Relationships 


centers  (LCC)  to  be  taken  off  alert  status  for  SELM  testing. 

The  JCS  could  maintain  their  confidence  that  the  reliability 
numbers  obtained  from  these  testing  programs  were  truly 
representative  of  the  systems  on  alert  status  because  an 
adequate  number  of  data  points  could  be  gathered.  Combining 
these  two  methods  also  gives  an  approximation  of  an  operational 
launch.  The  SELM  test  exercises  operational  ground  support 
equipment  and  missile  electronics  from  generation  of  the  launch 
command  up  to,  but  not  including,  booster  ignition.  Flight 
testing  utilizes  randomly  selected  missiles  from  the 
operational  force  that  are  removed  from  their  launch  facility, 
transported  to  Vandenberg  AFB,  and  launched  from  modified 
launch  facilities.  Combining  the  ground  test  (operational 
equipment  generating  signals  to  the  missile)  and  flight  test 
(missile  receiving  signals  and  launching)  provides  a  complete, 
if  broken,  sequence  of  events  that  is  representative  of  the 
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V.  • 


sequence  experienced  by  a  missile  launched  from  an  operational 
launch  facility. 

Department  of  Defense  (DOD)  Directive  5000.3  requires  that 
testing  be  as  conducted  in  a  manner  that  is  as  operationally 
realistic  as  possible  (6: 3; 7:17).  This  has  been  determined  by 
SAC  to  mean  SELM  and  flight  testing  (9;10;11).  This  strict 
interpretation  of  DOD  Directive  5000.3  was  appropriate  in  an 
era  of  adequate  flight  test  assets  and  the  ability  to  obtain 
SELM  data.  However  the  PEACEKEEPER  program  may  fall  short  in 
both  of  these  areas.  A  Congressional  Budget  Office  study  has 
determined  that  while  SAC  has  proposed  a  "modest"  flight  test 
program,  it  will  require  too  many  missile  launches  and  should 
therefore  be  reduced  (5:14,15).  The  simulated  electronic 
launch-PEACEKEEPER  (SELP)  program  will  be  scaled  down  from  the 
SELM  program  because  while  removing  11  MINUTEMAN  missiles  from 
alert  status  affects  only  1.1  percent  of  that  system,  removing 
11  PEACEKEEPERS  involves  22  percent  of  the  silo  deployed 
portion  of  that  system  -  an  unacceptably  large  percentage  (11). 
SAC  believes  that  they  have  proposed  a  program  using  the 
absolute  minimum  number  of  tests  (8,10).  Since  the  confidence 
in  the  ability  to  detect  a  degrade  in  the  system's  reliability 
is  currently  a  direct  result  of  the  number  of  tests  conducted, 
reducing  the  number  of  test  assets  below  the  minimum  number 
required  results  in  a  decrease  in  the  confidence  that  can  be 
placed  in  those  reliability  numbers. 


SAC  has  a  requirement  to  determine  the  reliability  of  the 
PEACKK KEPER  weapon  system.  They  are  faced  with  the  choice  of 


continuing  with  traditional  means  of  ICBM  testing  and 
potentially  not  having  test  assets  remaining  towards  the  end  of 
the  system's  life,  or  finding  ways  of  reducing  the  average 
yearly  requirement  for  the  number  of  test  flights.  There  are 
historical  precedents  for  both  alternatives.  The  TITAN  ICBM 
weapon  system  had  not  been  flown  in  an  operational  test  launch 
between  1969  and  its  deactivation  in  1987.  MINUTEMAN  II  has 
seen  a  reduced  number  of  flight  tests  in  recent  years  with  only 
eight  flight  tests  conducted  since  1980  (16).  Yet  the  TITAN 
system  was  still  included  in  the  SIOP  until  the  final  missile 
was  deactivated  and  MINUTEMAN  II  continues  to  be  part  of  the 
SIOP  (11). 

Statement  of  the  Problem 

The  Air  Force  must  determine  a  testing  strategy  for  the 
PEACEKEEPER  system.  Using  decision  analysis,  can  a  pilot  model 
be  developed  that  captures  the  essential  variables  and 
relationships  that  determine  what  this  strategy  will  be? 

Purpose  of  the  Study 

This  was  a  pilot,  or  preliminary,  decision  analysis 
because  of  the  limitations  stated  below.  Its  purpose  was  to 
present  a  viable  procedure  for  accomplishing  the  actual 
decision  by  identifying  and  analyzing  the  basic  issues  of  the 


Influence  diagrams  proved  to  be  useful  in  determining  the 


key  variables  and  relationships  between  the  variables  that 
affect  the  decision  making  process.  Analysis  of  the  model 
using  the  Performs  decision  analysis  software  program  provided 
answers  to  questions  that  a  decision  maker  would  want  to  know. 

The  pilot  model  provided  a  framework  from  which  a  full- 
scale  analysis  can  be  developed.  The  model  is  a  very 
simplified  representation  of  the  problem  and  is  mainly  useful 
for  determining  the  most  important  relationships.  The 
potential  for  decision  analysis  to  handle  such  a  problem  was 
established  and  a  model  was  developed  that  the  Air  Force  could 
use  in  further  understanding  this  decision  problem. 

Limitations  of  the  Study 

There  were  two  major  limitations  to  this  study: 

Restriction  from  use  of  classified  material.  This  thesis 
remained  unclassified.  This  was  a  limitation  because  the 
actual  reliability  numbers  gathered  from  ICBM  testing  are 
classified.  For  operations  security  reasons,  numbers  used  in 
this  thesis  were  unclassified  approximations  that  were  not 
related  to  the  actual  numbers.  Because  the  numbers  used  can 
make  a  difference  in  the  final  decision,  the  decision  derived 
from  this  study  cannot  be  strictly  applied  to  the  actual 
situation.  This  is  easily  alleviated  by  replacing  the 
unclassified  numbers  with  the  correct,  classified  data  and 
reanalyzing  the  data. 


Use  of  Substitute  Decision  Maker.  The  actual  decision 
maker  for  this  problem  would  have  been  a  general  officer, 
probably  the  Commander-In-Chief,  Strategic  Air  Command.  Not 
having  access  to  the  real  decision  maker  and  his  staff, 
information  was  gathered  from  many  officers  knowledgeable  in 
various  aspects  of  ICBM  acquisition,  operations,  testing,  and 
evaluation.  These  inputs  were  combined  into  a  simple  model 
that  the  author  believes  is  representative  of  the  current 
environment.  However,  each  decision  analysis  is  tailored  to  a 
specific  decision  maker.  As  soon  as  the  decision  maker 
changes,  the  analysis  must  be  reaccomplished  to  reflect  the  new 
decision  maker.  For  this  decision  analysis  to  be  usable,  the 
actual  decision  maker  would  have  to  be  used.  Additionally,  the 
actual  decision  maker  would  have  expert  resources  available  to 
provide  him  and  the  analyst  with  more  accurate  inputs  to  the 
decision  analysis  model. 

Definitions 

For  the  purposes  of  this  study,  the  following  terms  are 
defined : 

Decision;  A  decision  is  an  "irrevocable  allocation  of 
resources"  (9:23).  It  is  differentiated  from  an  outcome,  which 
is  a  state  that  occurs  as  a  result  of  what  decision  is'  made. 

The  decision  maker  can  make  a  decision  but  has  no  absolute 
control  over  what  the  outcome  of  the  decision  will  be. 

Decision  analysis:  "The  process  of  an  analyst  working  closely 
with  a  decision  maker  to  build  a  mathematical  model  relevant  to 


a  decision"  (14).  Its  purpose  Is  to  help  the  decision  maker 
better  understand  the  decision  problem  by  developing  a 
quantitative  model  to  represent  a  decision  problem.  The  model 
is  then  analyzed  mathematically  to  assist  the  decision  maker  in 
gaining  insight  and  understanding  Into  the  problem  (13:1-2). 
Decision  maker:  The  Individual  who  will  ultimately  make  the 
decision.  This  person  has  the  power  to  commit  the  organization 
to  a  course  of  action. 

Pilot  analysis;  "A  simplified,  approximate,  but  comprehensive, 
analysis  of  a  decision  problem"  (9:12).  It  helps  to  provide 
understanding  and  communications  about  the  problem  and  is  used 
as  an  aid  in  preparing  the  final  decision  analysis. 

Values :  Represent  the  desirability  of  each  outcome. 


Department  of  Defense  Directive  5000.3,  dated  12  Mar  86, 
states  that  "The  primary  purpose  of  all  T&E  (Test  and 
Evaluation)  is  to  make  a  direct  contribution  to  the  timely 
development,  production  and  fielding  of  systems  that  meet  the 
user's  requirements  and  are  operationally  effective  and 
suitable"  (6:2).  Air  Force  Regulation  (AFR)  80-14  allows  for 
combining  Developmental  Test  and  Evaluation  (DT&E)  and 
Operational  Test  and  Evaluation  (OT&E)  if  "separate  testing 
would  cause  . . .  increases  in  system  acquisition  costs  or  in 
resources"  (7:7).  Because  of  "the  limited  time  available  for 
testing,  fiscal  limitations,  and  because  ICBM  tests  are 
generally  destructive  in  nature"  a  combined  DT&E/OT&E  program 
has  been  established  for  the  PEACEKEEPER  weapon  system  (6:2). 

The  system's  test  cycle  is  composed  of  DT&E,  Initial 
Operational  Test  and  Evaluation  (IOT&E),  Follow-on  Operational 
Test  and  Evaluation  ( FOT&E )  (I),  and  FOT&E(2)  Phases  I  and  II. 
Each  testing  period  has  its  own  unique  objectives  depending  on 
the  maturity  of  the  system  and  each  agencies  charter.  SAC,  the 
Ballistic  Missile  Office  (BMO),  and  the  Air  Force  Operational 
Test  and  Evaluation  Center  (AFOTEC)  are  involved  in  the 
decision  on  how  an  ICBM  will  be  tested  but  their  influence 
varies  with  the  maturity  of  the  system.  Figure  2  summarizes 
the  proposed  test  program. 
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Figure  2.  Proposed  PEACEKEEPER  Test  Program 


a.  Assess  critical  issues. 

b.  Determine  how  well  the  contract  specifications  have  been 
met . 

c.  Identify  and  report  system  deficiencies. 

d.  Determine  system  compatibility  and  interoperability  with 
existing  and  planned  equipment  or  systems. 

e.  Report  reliability  in  relation  to  the  approved 
reliability  growth  plan,  and  to  estimate 
maintainability,  availability,  and  logistics 
supportability  of  the  system  at  maturity. 

f.  Certify  that  the  system  is  safe  and  ready  for  dedicated 
OT&E . 

g.  Validate  any  configuration  changes  caused  by  correcting 
deficiencies,  modifications,  or  product  improvements. 

h.  Assess  human  factors  and  identify  limiting  factors. 

i.  Assess  the  technical  risk  and  evaluate  compliance  with 
the  specifications,  in  relation  to  operational 
requirements  (including  reliability,  maintainability, 
and  availability),  life  cycle  costs,  and  schedules. 

j.  Determine  system  response  or  hardness  to  the  nuclear  and 
conventional  environments  in  order  to  support  system 
survivability  assessment  as  directed,  and  to  assess 
system  vulnerability,  including  hardness  features  and 
radloelectronic  combat  vulnerability. 

k.  Verify  the  accuracy  and  completeness  of  the  technical 
orders  developed  to  maintain  and  operate  the  weapon 
system. 

l.  Gather  Information  for  training  programs  and  technical 
training  materials  needed  to  support  the  weapon  system. 

m.  Provide  Information  on  environmental  issues  to  be  used 
in  preparing  environmental  Impact  assessments. 


n.  Determine  system  performance  limitations  and  safe 
operating  parameters  (7:7). 

AFSC  tasked  the  BMO,  headquartered  at  Morton  AFB  in  San 
Bernardino,  Ca.,  to  be  the  System  Program  Office  (SPO)  for 
PEACEKEEPER.  As  the  implementing  command  for  DT&E,  BMO : 


a.  Assigns  the  test  director  to  manage  DT&E. 

b.  Integrates  DT&E  and  OT&E  requirements  to  form  a  combined 
test  plan  to  accommodate  the  T&E  needs  of  the 
implementing  and  OT&E  commands.  The  combined  test  plan 
will  integrate  the  schedule  and  resource  requirements, 
but  will  not  affect  the  test  objectives  of  either  the 
developmental  test  plan  or  the  operational  test  plan. 

c.  Provides  the  test  articles  necessary  to  support  both  the 
DT&E  and  OT&E  portions  of  the  combined  test  program. 

d.  Arranges  for  support  resources  and  establishes  schedules 
necessary  to  carry  out  the  combined  test  program. 

e.  Reviews  and  approves  all  safety  aspects  of  the  combined 
test  program. 

f.  Provides  the  interface  with  the  system  contractor  (7:8). 
Although  there  is  no  distinct  break  point  between  DT&E  and 

I OT&E,  the  first  10  launches,  which  Initially  evaluated  the 
system  components  and  integrated  them  into  the  launch  facility, 
could  be  considered  DT&E  flights. 


Initial  Operational  Test  and  Evaluation  (IOT&E) 

I OT&E  is  conducted  in  an  environment  that  becomes 
increasingly  more  operationally  realistic  as  the  testing 
program  proceeds.  According  to  AFR  80-14,  "the  primary  purpose 
of  Air  Force  OT&E  is  to  ensure  that  only  operationally 
effective  and  suitable  systems  are  delivered  to  the  operating 
forces"  (7:7).  The  IOT&E  objectives  are  to: 

a.  Evaluate  the  operational  effectiveness  and  operational 
suitability  of  the  system. 

b.  Answer  unresolved  critical  operational  issues. 

c.  Identify  and  report  operational  deficiencies. 

d.  Recommend  and  evaluate  changes  in  system  configuration. 


e.  Provide  information  for  developing  and  refining: 

(1) .  Logistics  and  software  support  requirements  for  the 

system. 

(2) .  Training,  tactics,  techniques,  and  doctrine 

throughout  the  life  of  the  system. 

f.  Provide  information  to  refine  operation  and  support 

( O&S )  cost  estimates  and  identify  system  characteristics 
or  deficiencies  that  can  significantly  affect  O&S  costs. 

g.  Determine  if  the  technical  publications  and  support 
equipment  are  adequate. 

h.  Assess  the  survivability  of  the  system  in  the 
operational  environment  (7:8). 

Flights  11  through  16  may  be  considered  to  have  been  IOT&E 

flights.  They  consisted  of  launches  that  increasingly 

integrated  the  components  into  more  of  the  final  production 

configuration.  BMO  was  responsible  for  conducting  the  tests 

and  worked  closely  with  AFOTEC  to  incorporate  AFOTEC' s 

increasing  inputs  into  each  test's  objectives. 

AFOTEC,  as  the  OT&E  command,  has  an  increasing  interest  i 
the  test  program  through  F0T&E(1).  They  participate  in  the 
OT&E  process  through  the  completion  of  the  combined  test 
program  (7:3)  and: 

a.  Assign  the  test  director  and  deputy  test  director  to 
conduct  OT&E. 

b.  Prepare  the  OT&E  plan  for  Integration  into  the  combined 
plan. 

c.  Coordinate  on  the  combined  test  plan. 

d.  Arrange  for  OT&E  peculiar  resources. 

e.  Conduct  and  report  on  the  OT&E,  and  make  an  independent 
operational  evaluation  (7:8). 


Follow-on  Operational  Test  and  Evaluation  ( FOT&E ) 

FOT&E  began  after  the  first  operational  missile  was 


deployed.  This  final  testing  period  is  separated  into  three 
catagories:  FOT&E(l)  will  continue  to  be  conducted  by  BMO  and 
closely  aligned  with  AFOTEC  objectives  (BMO  flight  test 


missiles  17-20);  FOT&E(2)  Phase  I  is  scheduled  to  begin  in  1989 
when  SAC  begins  to  launch  missiles  that  have  previously  been 
operationally  deployed;  and  FOT&E(2)  Phase  II.  FOT&E(l)  began 
with  BMO  flight  17,  the  first  launch  after  an  operational 
missile  had  been  deployed.  The  remaining  three  flights  will 
take  place  under  BMO  control  with  AFOTEC  and  SAC  participation, 
through  1989.  FOT&E(2)  Phase  I  testing  for  the  silo  deployed 
missiles  is  planned  to  begin  in  1989  and  will  consist  of  SAC 
conducting  24  launches  over  a  three  year  period  (10).  Although 
the  objectives  and  responsibilities  for  conducting  this  testing 
are  not  specifically  listed  as  are  those  for  the  previous 
phases,  this  testing  is  designed  to  assess  the  deployed  system 
accuracy  and  reliability  and  provide  a  data  base  of  SIOP 
planning  factors  (2).  Plans  for  Phase  I  testing  for  rail 
garrison  deployed  missiles  have  not  been  finalized  (10).  The 
rail  garrison  Phase  I  may  also  have  the  additional  objective  of 
determining  whether  the  missiles  from  the  different  basing 
modes  can  be  regarded  as  having  similar  characteristics  and  can 
therefore  be  regarded  as  coming  from  the  same  population,  or  if 
they  are  different  and  two  separate  testing  programs  must  be 
developed  ( 2 ) . 

Phase  II  testing  begins  after  the  completion  of  Phase  I 
and  continues  for  the  deployed  life  of  the  system.  It  is 
designed  to  update  SIOP  planning  factors  and  to  provide  a  means 
of  detecting  system  degrades  (2).  If  the  results  of  Phase  I 
testing  Indicate  that  the  missiles  from  the  two  deployments 
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come  from  the  same  population.  Phase  II  is  scheduled  to  consist 
of  84  launches.  If  the  results  indicate  significant 
differences  in  performance  between  the  missiles  from  the 
different  basing  modes,  further  assessment  will  be  made  as  to 
how  many  additional  test  assets  will  be  required  (10,12). 


Figure  3.  Top  Level  Organizational  Relationships 

All  these  agencies  are  participants  in  the  formulation  of 
a  test  strategy.  Given  that  the  Executive  branch,  OSD,  and  HQ 
USAF  have  supported  the  concept  of  acquiring  the  PEACEKEEPER 
system  and  have  provided  for  the  appropriate  funding  levels  in 
the  proposed  budgets.  Congress  must  still  appropriate  the 
monies.  BMO,  SAC,  and  AFOTEC  work  together  to  determine  the 
best  test  strategy  based  on  this  funding  level.  During  early 
DT&E  testing,  BMO  has  most  of  the  responsibility  for 
formulating  test  objectives,  with  SAC  and  AFOTEC  overseeing  the 
tests.  As  the  system  matures  and  the  test  articles  begin  to 
more  closely  resemble  the  final  deployment  configuration, 

AFOTEC  becomes  more  Involved  with  formulating  test  objectives 
to  verify  the  system  while  BMO  maintains  responsibility  for  the 
testing.  SAC  has  some  input  into  the  test  objectives  but 


mainly  observes  and  analyzes  the  test  results  for  their 
operational  Implications.  When  SAC  assumes  responsibility  for 
testing  the  deployed  force  at  FOT&E(2)  Phase  I,  they  control 
the  tests  and  BMO  and  AFOTEC  have  their  role  reduced  to 
developing  and  testing  system  modifications. 

SAC,  through  the  549th  Weapon  System  Evaluation  Squadron 
( WSES ) ,  determines  how  many  test  launches  are  required  to  meet 
Weapon  System  Evaluation  Guide  (WSEG)  92D,  DoD  Directive 
5000.3,  and  AFR  80-14  guidance.  By  using  the  equations: 

P= 1- ( 1-DR )  exp  n  (1) 

or 

n=[ln(l-p) ]/[ln(l-DR) ]  (2) 

where 

P  Is  the  confidence  of  detecting  the  failure  mode 
D  is  the  percentage  of  degrade  In  reliability 
caused  by  the  failure  mode 
R  Is  the  initial  ( nondegraded )  reliability 
n  is  the  number  of  missiles 

the  549th  WSES  determines  how  many  missiles  must  be  launched  in 
order  to  comply  with  JCS  guidance  that  requires  a  "stated 
degree  of  confidence  of  detecting  failure  modes  that  degrade 
reliability  by  certain  amounts"  (3).  A  confidence  of  .90  and  a 
degrade  of  .10  was  used  in  the  Congressional  Budget  Office 
report  and  in  the  WSES  30  Nov  1982  Memo  For  Record  (5:8;1:1). 
The  actual  confidence  and  degrade  probabilities  are  classified. 


The  following  assumptions  are  required  to  be  made  in 


order  to  use  these  equations: 

1)  The  failure  mode  of  interest  is  always  masked  by  other 
failures . 

2)  All  failures  of  the  system  are  detectable  by  analysts. 

3)  The  new  failure  mode  is  independent  of  the  original  failure 
mode . 

4)  The  test  size  is  based  on  the  ability  to  detect  at  least  one 
of  the  failure  modes. 

5)  The  degrade  in  reliability  is  assumed  to  occur 
instantaneously  (4:1). 

Current  testing  guidelines  do  not  specify  the  time  period 
over  which  the  confidence  of  detecting  a  degrade  is  required, 
therefore,  the  interval  for  detecting  the  failure  mode  may  be 
from  one  to  several  years.  This  being  the  case,  an  "n"  of  21 
can  be  launched  all  in  one  year,  giving  a  launch  rate  of  21 
missiles  per  year,  or  spread  over  three  years,  giving  a  launch 
rate  of  seven  missiles  per  year.  The  number  of  years  that  are 
required  to  accumulate  the  desired  confidence  of  detecting  a 
degrade  is  determined  from  "n"  and  the  yearly  launch  rate.  By 
multiplying  the  number  of  years  the  system  is  expected  to  be 
fielded  with  the  yearly  launch  rate,  the  total  number  of  test 
assets  required  for  the  life  of  the  system  can  be  determined. 

This  determination  is  presented  through  HQ  SAC/DOMV  to  the 
HQ  SAC/DO,  LG,  and  XP  staffs  for  their  coordination  as  depicted 
in  Figure  4.  Their  coordinated  position  is  presented  to  the 
Commander-in-Chlef ,  Strategic  Air  Command  and  he  gives  the 
final  approval  (16).  This  decision  on  test  levels  is  relayed 
through  HQ  SAC/XPQ  to  the  BMO  and  HQ  USAF/AQQ .  If  AQQ  accepts 
these  levels  for  Inclusion  in  the  PMD,  they  are  then  Included 


in  the  budget  submission  to  Congress  and  BMO  begins  the 


necessary  contractual  efforts  to  obtain  the  missiles, 
predicated  on  Congressional  funding.  Congress  can  accept  these 
levels  by  providing  full  funding  or  can  fund  at  a  different 
level.  If  this  level  is  lower  than  that  requested  by  the  Air 
Force,  SAC  must  return  and  ask  for  additional  assets  or  adjust 
their  test  program  to  work  within  the  Congressionally  mandated 
constraints.  The  funding  request  for  missile  assets  does  not 
occur  only  in  one  year,  but  is  spread  over  several  years.  The 
requests  are  based  on  the  number  of  required  missiles,  what  BMO 
believes  to  be  an  economical  buy  rate,  and  how  many  missiles 
have  previously  been  funded. 


remaining  12  year  expected  life  of  the  system.  If  an 


additional  50  operational  PEACEKEEPER  missiles  are  deployed  in 
the  rail  garrisoned  basing  mode,  SAC  would  like  to  conduct  an 
FOT&E ( 2 )  Phase  I  consisting  of  24  missiles  to  be  launched  over 
three  years  in  order  to  determine  if  there  is  a  difference  in 
the  missiles  from  the  two  basing  modes.  If  no  difference  is 
found,  the  testing  would  resume  with  results  being  applied  to 
both  systems.  If  a  difference  is  found,  separate  programs 
would  have  to  be  conducted  to  test  each  system.  The  current 
PMD  only  provides  for  an  additional  12  missiles  (12). 

With  this  understanding  of  the  overall  testing  philosophy 
and  the  agencies  responsible  for  formulating  the  test 
philosophies,  the  model  is  now  discussed. 


III.  Methodolo 


The  model  was  developed  based  on  Inputs  from  many  Air 
Force  officers  having  current  or  past  experience  In  one  or  more 
of  the  fields  concerning  ICBM  requirements  generation, 
operations,  testing,  evaluation,  and  acquisition.  These  inputs 
were  meshed  with  the  author's  four  years  of  experience  in 
MINUTEMAN  III  operations,  three  years  experience  at  Vandenberg 
AFB  in  both  MINUTEMAN  and  PEACEKEEPER  testing,  and  three  years 
at  Detachment  2,  Headquarters  Strategic  Air  Command  assigned  to 
the  Ballistic  Missile  Office  at  Norton  AFB.  In  these 
assignments  he  has  had  the  responsibility  for  generating  and 
articulating  SAC  requirements  to  the  BMO  and  its  contractors  as 
well  as  monitoring  the  developmental  progress  of  the 
PEACEKEEPER  and  Small  ICBM  systems.  The  model  resulted  from 
the  interactions  depicted  in  Figure  5. 


Figure  5.  Thesis  Development 


The  author  began  by  conducting  telephone  conversations 
with  officers  currently  assigned  to  HQ/SAC  XPQM  and  DOMV, 
Detachment  2  HQ/SAC  and  the  549th  WSES.  The  current  status  of 
the  PEACEKEEPER  acquisition  and  deployment  in  both  the  silo  and 
rail  garrison  basing  modes  was  obtained.  Issues  surrounding 
the  determination  of  the  number  of  missiles  to  deploy,  various 
basing  mode  concepts,  and  the  number  of  missiles  that  are 
required  to  conduct  a  test  program  were  reviewed.  The  current 
status  as  well  as  projections  for  the  future  of  the  MINUTEMAN 
test  program  were  discussed.  Parallels  were  drawn  between  the 
test  programs  for  the  TITAN,  MINUTEMAN  II,  MINUTEMAN  III,  and 
PEACEKEEPER  programs  in  light  of  past  test  policies,  current 
numbers  of  MINUTEMAN  II  and  MINUTEMAN  III  test  assets, 
projected  requirements  for  MINUTEMAN  II  and  MINUTEMAN  III 
assets,  stated  and  proposed  requirements  for  PEACEKEEPER  test 
assets  for  both  basing  modes,  current  and  projected  funding  for 
PEACEKEEPER  test  assets,  and  the  contractual  and  projected 
lifetime  of  the  PEACEKEEPER  system. 

Unclassified  regulations  were  reviewed  to  establish  the 
basis  for  how  and  why  testing  occurs. 

Finally,  three  instructors  from  the  Air  Force  Institute 
of  Technology  (AFIT)  faculty  each  separately  agreed  to  assume 
the  role  of  the  decision  maker  for  whom  the  model  would  be 
built.  The  officers  represented  a  variety  of  barkgrounds  and 
experiences  in  working  with  ICBMs,  with  one  having  been 
stationed  at  the  BMO  working  in  systems  acquisition,  another 
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having  HQ  SAC  experience,  and  the  third  having  been  assigned  to 
an  operational  xCBM  unit  as  a  missile  crew  member  and  later  as 
a  missile  maintenance  officer.  This  last  officer  was  also 
assigned  to  AFOTEC  and  participated  in  the  development  of  the 
first  PEACEKEEPER  test  plan. 

Each  officer  was  asked  to  take  a  top  level  view  of  the 
test  program  as  if  he  were  making  the  Air  Force  decision  on 
what  the  test  policy  should  be.  No  restrictions  were  placed  on 
the  options  which  could  be  considered  and  the  officers  were 
especially  encouraged  to  develop  alternatives  to  flight 
testing.  The  potential  outcomes  for  each  option  and  the  test 
program  as  a  whole  were  considered  and  the  question  of  why  have 
a  test  program  at  all  was  discussed.  The  field  of  decision 
makers  was  then  narrowed  to  one  officer  based  on  the  more  rapid 
development  of  his  model.  Since  his  model  was  based  on  the 
goal  of  meeting  the  confidence  levels  set  forth  in  DoD 
Directive  5000.3,  the  model  grew  to  represent  a  better  way  of 
determining  how  many  flight  test  missiles  are  needed  to  meet 
this  guidance  as  opposed  to  discovering  alternatives  or 
complements  to  flight  testing. 

More  conversations  were  held  with  various  members  of  the 
549th  VSES  in  an  effort  to  more  fully  understand  their  role  in 
developing  the  number  of  flight  test  missiles  and  the  equations 
they  use. 

The  model  was  developed  based  on  the  Inputs  from  each  of 


the  decision  makers,  information  obtained  from  officers 


assigned  to  SAC,  and  the  author's  experience  and  judgement. 

The  last  decision  maker  was  again  contacted  for  his  estimations 
of  the  abilities  of  launch  analysts  to  identify  failure  modes 
based  on  the  various  combinations  of  launches  and  failures 
associated  with  a  range  of  zero  to  three  missiles.  The  author 
assigned  probabilities  to  the  two  degrees  of  degrade  that  were 
present  in  the  model  and  then  assigned  values  to  all  the 
possible  outcomes.  The  emphasis  was  on  developing  a  model  that 
is  structurally  correct.  True  values  and  probabilities  can  be 
obtained  from  the  decision  maker  as  the  pilot  model  is  expanded 
into  a  full-scale  analysis. 

The  model  was  analyzed  using  the  software  program 
Per  forma,  version  1.0.  A  worth  lottery  was  calculated  and 
stochastic  sensitivity  analysis  conducted.  The  model  was  then 
expanded  to  Include  three  degrees  of  degrade  and  the  possible 
number  of  launches  was  expanded  to  eight.  Estimations  of  the 
analysts'  abilities  to  determine  failure  modes  were  expanded  by 
the  author  based  on  those  previously  given  by  the  decision 
maker.  A  new  set  of  values  was  assigned  by  the  author.  The 
expanded  model  was  first  analyzed  in  order  to  determine  the 
number  of  launches  per  year  that  gave  the  maximum  value. 

Various  other  analyses  were  then  performed  to  determine 
probability  distributions  that  might  prove  of  interest  to  a 


IV.  Model  Structure 


Overview 

A  format  was  needed  that  would  show  not  only  the  various 
pieces  of  Information  that  are  needed,  but  would  also  show  how 
the  pieces  relate  to  each  other.  The  use  of  influence  diagrams 
accomplished  this.  The  resulting  model,  shown  in  Figure  6, 
consists  of  a  decision  variable  (NOMSLS),  three  random  variable 
( DEGRADE%,  DETECT,  and  FAILURES),  a  value  variable  (V),  and 
arcs  between  the  variables  indicating  their  relationships.  The 
value  (V)  of  the  test  program  is  dependent  on  the  number  of 
missiles  launched  (NOMSLS),  the  percentage  of  the  force  that  is 
degraded  (DEGRADE*),  and  the  ability  of  the  analysts  to  detect 
the  degrade  mode  (DETECT).  Missiles  are  limited  in  quantity, 
may  cost  up  to  $98  million  each,  and  are  expended  upon  being 
launched  (10).  This  is  a  large  price  to  pay,  so  a  program  that 
requires  the  fewest  number  of  launches  is  preferred.  The  pilot 
model  only  covers  a  time  period  of  one  year.  This  time  period 
can  be  lengthened  or  compressed  as  necessary. 


The  DEGRADE*  effects  the  value  of  the  test  program  in 
that  large  degrades  are  more  important  to  discover  than  are 
small  degrades.  A  decision  maker  might  be  less  willing  to 
expend  valuable  test  assets  to  uncover  a  five  percent  degrade 
than  he  would  to  discover  a  fifty  percent  degrade.  Larger 
degrades  may  also  be  easier  to  detect  because  they  should  be 
observed  more  often. 

A  missile  that  fails  is  of  value  only  if  the  cause  of  the 
failure  can  be  discovered.  Otherwise,  it  only  represents  a 
worry.  The  ability  of  the  analysts  to  DETECT  the  cause (s)  of 
failures  is  a  significant  Input  into  the  value  of  any  test 
program.  Analysts  are  not  perfect  and  neither  is  the  equipment 
that  gathers  the  data  for  them.  This  model  does  not  assume 
that  every  failure  will  be  identified  with  100  percent 
accuracy. 

The  ability  to  detect  a  degrade  mode  is  dependent  on  the 
number  of  missiles  launched,  the  number  of  failures  that  occur, 
and  the  number  of  tests  that  result  in  failures.  The  more 
missiles  launched,  the  greater  should  be  the  probability  of 
launching  one  or  more  missiles  that  have  the  failure  mode 
present . 

The  number  of  failures,  then,  is  dependent  on  the  degree 
to  which  the  force  is  degraded  and  the  number  of  missiles 


launched . 


The  decision  variable  (NOMSLS  or  number  of  missiles) 
represents  the  decision  maker's  choice  of  how  many  test 
missiles  to  launch.  It  contains  the  alternatives  of  launching 
0,  1,  2,  3,  4,  5,  6,  7,  or  8  missiles  per  year.  A  maximum  of 
eight  flights  that  can  be  launched  from  Vandenberg  AFB  each 
year  without  imposing  severe  constraints  upon  resources  at  both 
Vandenberg  AFB  and  F.E.  Warren  AFB  (10;12).  Arcs  from  NOMSLS 
to  V,  FAILURES,  and  DETECT  indicate  that  these  variables  are 
dependent  on  the  choice  of  NOMSLS. 

The  random  variable  DEGRADE%  represents  the  proportion  of 

the  force  that  is  degraded  by  the  failure  mode.  It  contains  the 

possibilities  of  2%,  10%,  or  50%  (.02,  .10,  or  .50)  of  the 

force  being  degraded.  The  probabilities  for  these  states 

occurring  were  set  in  accordance  with  the  following  values: 

Outcome  .02  .10  .50 

Probability  .50  .30  .20 

of  occurrence 

The  outcomes  were  chosen  by  the  author  based  on  various  levels 
of  concern  with  failure  modes  being  present  in  the  missile 
force.  The  examples  in  both  the  Congressional  Budget  Office 
report  and  the  549th  WSES  memos  use  a  10%  degrade,  hence  the 
choice  of  that  number  (5:8;1:1).  A  number  below  this  threshold, 
.02,  was  chosen  so  as  to  observe  the  effects  of  having  a 
relatively  insignificant  failure  mode  while  .50  was  chosen  so 
as  to  observe  the  effects  of  having  a  very  significant  failure 
mode  present.  The  probabilities  of  occurrence  were  assigned  in 
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the  belief  that,  given  the  historical  reliability  of  missile 
systems,  smaller  degrades  are  more  probable  than  larger  ones. 

It  would  be  easy  to  replace  these  outcomes  and  probabilities 
with  others  as  directed  by  a  more  knowledgeable  decision  maker 
or  his  experts. 

The  random  variable  FAILURES  represents  the  number  of  test 
missiles  that  fail.  It  contains  the  possible  outcomes  of  0,  1, 
2,  3,  4,  5,  6,  7,  and  8.  The  variable  shows  the  probability  of 
the  number  of  failures  occurring  dependent  on  the  percentage  of 
the  force  that  is  degraded  (from  DEGRADE%)  and  the  number  of 
missiles  that  are  launched  (from  NOMSLS).  For  this  pilot 
model,  the  number  of  failures  was  modeled  as  a  binomial  random 
variable  with  "n"  being  the  number  of  missiles  launched 
(NOMSLS)  and  "pH  being  the  proportion  of  the  force  degraded  by 
a  failure  mode  (DEGRADE*). 

The  random  variable  DETECT  describes  the  probability  that 
an  analyst  will  identify  a  failure  mode.  This  ability  is 
conditioned  on  NOMSLS,  FAILURES,  and  DEGRADE*.  See  Appendix  A. 

The  value  variable  describes  the  relative  goodness  of  each 
of  the  possible  outcomes.  It  is  based  on  the  concept  of  having 
a  nominal  90  percent  alert  rate  for  the  force  and  the  decision 
maker's  willingness  to  accept  some  lower  alert  rate  for  a  one 
week  period  in  order  to  have  the  opportunity  to  run  the  test 
program.  As  examples,  a  value  of  5  means  that  the  decision 
maker  would  be  willing  to  have  an  85  percent  (90  -  5)  alert 
rate  for  one  week.  A  value  of  -3  means  that  the  decision  maker 


would  want  his  alert  rate  to  be  93  (90  -  (-3)),  an  increase 
above  the  nominal,  before  he  would  allow  the  testing  to  take 
place.  These  values  were  chosen  by  the  author  only  as  an 
attempt  to  show  differentiation  between  the  desirability  of  the 
possible  outcomes.  The  values  are  not  needed  for  model  to 
answer  many  of  the  questions  of  interest  to  a  decision  maker. 
They  are  easily  replaced  by  a  value  determination  method  that  a 
decision  maker  might  find  more  preferable.  The  values  are 
listed  in  Appendix  B. 

This  structure  outlines  the  basic  decision  analysis 
problem,  identifying  the  Important  variables  and  showing  their 
relationships.  Now  it  is  possible  to  analyze  the  model  in 
order  to  answer  questions  that  are  important  to  the  decision 
maker . 
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V.  Results 


The  specific  findings  in  this  chapter  are  only  useful  for 
illustrating  the  kind  of  information  that  this  model  can 
provide  to  a  decision  maker.  They  should  not  be  considered 
solutions  in  and  of  themselves  because  the  data  used  in  the 
pilot  model  are  only  approximations  of  the  actual  data.  What 
is  important  is  the  kind  of  information  that  this  model  can 
provide.  The  current  model  shows  many  important  relationships. 
The  substance  of  these  relationships  will  take  on  even  more 
meaning  when  actual  data  is  used. 


Maximum  Expected  Value 

The  model  was  analyzed  to  obtain  the  maximum  expected 
value.  Figure  7  uses  influence  diagrams  to  illustrate  the 
steps  used  to  obtain  the  maximum  expected  value.  Removing  each 
random  variable  into  the  value  node,  which  corresponds  to 
mathematical  expectation,  left  the  value  dependent  on  the 


number  of  missiles  to  be  launched  as  shown  in  Table  1. 


Table  1.  Maximum  Expected  Value 
MOMSLS  V 


0 

1 

2 

3 

4 

5 

6 

7 

8 


-3.166 

1.2377 

0.254 

1.2197 

1.711 

1.9083 

1.9679 

1.9113 

1.7598 


The  decision  maker  would  have  the  greatest  value  for  a  program 
that  launches  six  missiles  per  year.  However,  It  Is  noted  that 
launching  other  than  zero  or  two  missiles  gives  values  that 
range  over  a  distance  of  only  0.7482.  This  indicates  that  more 


data  might  be  needed  in  order  to  arrive  at  the  best  decision. 


Figure  7.  Steps  to  Calculate  Maximum  Expected  Value 


Figure  8  depicts  how  the  model  can  be  further  analyzed  to 


find  the  expected  value  based  on  the  percentage  of  the  force 
that  is  believed  to  possess  the  degrade.  Table  2  shows  that 
launching  zero  missiles  gives  the  largest  expected  value  if  the 


degrade  is  believed  to  affect  only  two  percent  of  the  force. 


Figure  8.  Expected  Value  Dependent  on  DEGRADE* 


Table  2.  Expected  Values  For  2  Percent  Degrade 


i 

t 


B 


\ 


DEGRADE* 

NOMSLS 

V 

.02 

0 

-0.97 

.02 

1 

-1.8736 

.02 

2 

-9.3053 

.02 

3 

-9.1891 

.02 

4 

-8.9765 

.02 

5 

-8.8823 

.02 

6 

-8.7977 

.02 

7 

-8.722 

.02 

8 

-8.6552 

A  degrade  affecting  a  larger  portion  of  the  force  shows 
different  values,  as  listed  in  Table  3. 


Table  3.  Expected  Values  For  10  Percent  Degrade 
DEGRADE* _ NOMSLS _ V 


.10 

0 

-2.77 

.10 

1 

-3.585 

.10 

2 

-4.3862 

.10 

3 

-3.9524 

.10 

4 

-3.385 

.10 

5 

-3.1743 

.10 

6 

-3.1973 

.10 

7 

-3.2763 

.10 

8 

-3.3928 

While  launching  zero  missiles  still  gives  the  largest  expected 
value,  the  range  of  values  shows  a  difference  of  only  1.6162, 
again  possibly  indicating  that  there  is  not  much  difference 
between  the  options  and  that  other  data  may  be  valuable  in 
formulating  the  final  decision.  Finally,  a  degrade  affecting  a 
large  percentage  of  the  force  gives  the  expected  values  listed 
in  Table  4. 


Table  4.  Expected  Values  For  50  Percent  Degrade 


DEGRADE* 

NOMSLS 

V 

.50 

0 

-9.25 

.50 

1 

16.25 

.50 

2 

31.1125 

.50 

3 

35.0 

.50 

4 

36.0737 

.50 

5 

36.5087 

.50 

6 

36.6298 

.50 

7 

36.2759 

.50 

8 

35.526 

The  largest  expected  value  is  obtained  with  six  launches, 
however  the  range  of  values  between  three  and  six  launches  Is 
only  1.6298.  As  in  the  other  cases,  more  data  may  be  helpful. 

With  either  a  five  or  ten  percent  degrade,  launching  zero 
missiles  gives  the  maximum  expected  value,  which  in  both  cases 
is  negative.  The  possible  existence  of  a  50  percent  degrade 
dominates  the  outcomes  for  the  complete  model's  maximum 
expected  value  and  the  number  of  missiles  that  should  be 
launched . 

Sensitivity  of  Failure  to  DEGRADE* 

The  probability  of  a  failure  occurring  is  dependent  on  the 
proportion  of  the  force  that  is  degraded  ( DEGRADE% )  and  the 
number  of  missiles  launched  (NOMSLS).  Table  5  lists  these 
probabilities . 


Table  5.  Probability  of  Failure 


Probability  of  at  Least  one  Failure  Occurring 

Table  6  lists  the  probability  of  at  least  one  failure 
occurring/  given  the  probability  distribution  of  any  of  the 
degrade  percentages  being  present. 


Table  6.  Probability  of  At  Least  One  Failure  Occurring 

N  Probability  of  seeing 

at  least  one  failure 

0  0.0 

1  0.14 

2  0.2268 

3  0.2857 

4  0.32947 

5  0.36466 

6  0.39456 

7  0.4209 

8  0.44467 


Launching  the  six  missiles  indicated  by  the  maximum  expected 
value  provides  less  than  a  40  percent  chance  of  seeing  at  least 
one  failure.  Launching  the  largest  possible  number  of  missiles 
per  year  provides  less  than  a  45  percent  chance  of  seeing  at 
least  one  failure.  Figure  9  shows  how  these  probabilities  were 
der lved . 


Figure  9.  Probability  of  Failure  Occurring 
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If  the  decision  maker  believes  that  the  analysts  will  not 
be  able  to  discover  the  cause  of  failure,  the  model  gives  the 
greatest  expected  value  of  -3.4  to  launching  zero  missiles.  If 
the  analysts  will  always  discover  the  failure  mode,  launching 
one  missile  gives  the  greatest  expected  value  of  41.4317. 
However,  analysts  tend  not  to  be  perfectly  good  or  bad.  The 
probabilities  of  detecting  the  failure  mode  based  on  the  number 
of  missiles  launched  are  listed  in  Table  7.  Figure  10  depicts 
the  steps  taken  in  order  to  obtain  these  probabilities. 

Table  7.  Analyst's  Probability  of  Detecting  the  Failure  Mode 

PROBABILITY  OF  DETECTION 


NOMSLS 

NO 

YES 

0 

.99 

.01 

1 

.8777 

.1223 

2 

.7866 

.2134 

3 

.7268 

.2732 

4 

.6951 

.  3049 

5 

.6758 

.  3242 

6 

.6593 

.3407 

7 

.  6453 

.  3547 

8 

.  6329 

.  3671 

Launching  six  missiles  means  that  there  is  only  a  34  percent 
chance  of  detecting  a  failure  mode.  Eight  launches  improves 
this  by  less  than  three  percent. 


Figure  10.  Probability  of  Detecting  Failure  Mode 
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Detection  Dependent  on  Degrade 


The  probability  of  detecting  a  failure  mode  depends  on  the 
percentage  of  the  force  that  is  degraded,  as  shown  in  Table  8. 

Table  8.  Probability  of  Detecting  a  Failure  Mode 


NOMSLS 

DETECT 

DETECT 

DETECT 

at  .02 

at  .  10 

at  .  50 

0 

.01 

.01 

.01 

1 

.03 

.11 

.38 

2 

.05 

.20 

.63 

3 

.07 

.29 

.75 

4 

.09 

.33 

.79 

5 

.10 

.  36 

.83 

6 

.11 

.38 

.86 

7 

.12 

.40 

.89 

8 

.12 

.41 

.91 

Launching  six  missiles  provides  an  11  percent  confidence  of 
detecting  a  two  percent  degrade,  a  38  percent  confidence  of 
detecting  a  ten  percent  degrade,  and  an  86  percent  confidence 
of  detecting  a  50  percent  degrade.  If  eight  missiles  are 
launched  in  the  year,  there  is  only  a  41  percent  probability  of 
detecting  a  ten  percent  degrade.  A  90  percent  confidence  of 
detecting  a  degrade  can  be  achieved  over  a  one  year  period  only 
if  eight  missiles  are  launched  and  half  the  force  is  affected 
by  the  degrade. 

Probability  of  DEGRADE*  Based  on  FAILURES 
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failed.  Table  9  shows  the  DEGRADE*  distribution  dependent  on 
the  number  of  missiles  launched  and  the  number  of  failures  that 
occur . 


Table  9.  Probability  That  Level  of  Degradation  Exists 


PROBAB I L I TY DEGRADE% 


NOMSLS 

FAILURES 

.02 

.  10 

.50 

0 

0 

.5 

.3 

.2 

1 

0 

.57 

.31 

.12 

1 

1 

.07 

.21 

.72 

2 

0 

.62 

.31 

.07 

2 

1 

.11 

.31 

.58 

2 

2 

.00 

.06 

.94 

3 

0 

.66 

.31 

.03 

3 

1 

.16 

.41 

.  43 

3 

2 

.01 

.10 

.89 

3 

3 

.00 

.01 

.99 

4 

0 

.69 

.29 

.02 

4 

1 

.21 

.50 

.29 

4 

2 

.01 

.16 

.83 

4 

3 

.00 

.02 

.98 

4 

4 

.00 

.02 

.98 

5 

0 

.71 

.28 

.01 

5 

1 

.26 

.56 

.18 

5 

2 

.02 

.25 

.73 

5 

3 

.00 

.04 

.96 

5 

4 

.00 

.01 

.99 

5 

5 

.  00 

.  00 

1 . 00 

6 

0 

.73 

.  26 

.  01 

6 

1 

.  30 

.  59 

.11 

6 

2 

.  03 

.  37 

.60 

6 

3 

.00 

.07 

.93 

6 

4 

.00 

.  01 

.99 

7 

5 

.00 

.01 

.99 

7 

6 

.  00 

.00 

1.00 

7 

7 

.  00 

.00 

1.00 

8 

0 

.76 

.23 

.01 

8 

1 

.36 

.60 

.04 

8 

2 

.07 

.62 

.31 

8 

3 

.01 

.  18 

.81 

9 

4 

.  00 

.02 

.98 

8 

5 

.00 

.01 

.99 

8 

6 

.00 

.00 

1 . 00 

8 

7 

.00 

.00 

1.00 

8 

8 

.00 

.00 

1.00 

raw 


VI .  Recommendations 

This  pilot  analysis  should  be  used  as  a  basis  for 
developing  a  complete  decision  analysis  as  an  aid  in 
determining  the  appropriate  test  strategy  for  the  PEACEKEEPER 
weapon  system.  To  become  a  full-scale  decision  analysis  model 
and  become  a  factor  in  the  decision  making  process,  the  value 
formulation  must  be  redefined  in  terms  that  the  decision  maker 
believes  appropriate,  accurate  probabilities  must  be  identified 
and  assigned  to  the  variables,  and  the  model  must  be  expanded 
to  include  all  relevant  variables  of  the  decision  problem. 

Value  Redefinition 

A  better  method  of  determining  the  value  of  an  strategy 
based  on  the  expected  outcomes  needs  to  be  identified  and 
incorporated  into  the  model.  The  use  of  a  willingness  to  give 
up  a  percentage  of  the  force  alert  rate  for  a  period  of  time  in 
order  to  detect  a  failure  mode  is  a  simple  measurement,  but  it 
may  not  be  realistic.  Starting  from  an  assumed  90  percent 
alert  rate  only  allows  for  a  minimum  value  of  -10.  This 
restricts  the  range  of  values  to  which  an  undesirable  outcome 
can  be  assigned  because  there  cannot  be  an  alert  rate  higher 
than  100  percent.  For  example,  the  value  for  launching  three 
missiles  and  not  being  capable  of  detecting  the  failure  mode  is 
the  same,  -10,  (meaning  that  the  alert  rate  would  have  to  be 
raised  to  100  percent  before  that  outcome  would  be  equivalent 
to  all  others)  Independent  of  the  percentage  of  the  force  that 
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is  degraded.  However,  it  is  worse  to  not  detect  the  failure 
mode  in  a  force  that  is  50  percent  degraded  than  to  not  detect 
it  in  a  force  that  is  only  degraded  by  two  percent.  Thus,  the 
two  outcomes  are  not  equal.  Discovering  an  accurate  method  of 
determining  the  values  of  test  strategies  may  prove  to  be  a 
difficult  task.  Values  based  on  the  cost  of  maintaining  the 
degraded  portion  of  the  force  and  the  cost  of  the  test  program 
needed  to  discover  the  degradation  may  provide  more  useful 
information . 


Probability  Assessments 

Estimates  concerning  the  true  ability  of  analysts  to 
discover  failure  modes  should  be  obtained  and  incorporated. 

This  could  be  done  in  conjunction  with  the  expansion  of  the 
model.  The  different  failure  modes  can  be  broken  out  of  the 
FAILURE  variable  and  separate  variables  developed  for  each 
failure,  group  of  failures,  or  combination  of  failures. 

Partial  failures  can  be  included  as  can  the  possibility  that  a 
failure  mode  might  be  masked  by  other  failure  modes.  The 
analysts'  abilities  to  both  identify  an  existing  failure  mode 
and  not  to  identify  a  mode  that  does  not  exist,  can  be 
incorporated  into  the  model  against  each  of  the  listed  failures 
Instead  of  as  an  overall  average.  This  should  give  much  better 
determinations  of  the  ability  to  detect  degrades  because  this 
ability  should  vary  depending  on  the  type,  severity,  and  number 
of  degrades  present.  Discovering  these  estimates  may  be  a  time 


consuming  task  depending  on  how  the  level  of  failures  are 
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expanded.  Influence  diagrams  can  be  of  help  in  determining 
what  estimates  are  needed.  1st  Strategic  Aerospace  Division, 
Test  Analysis  Division  (1  STRAD/TOH)  may  help  provide  the 
needed  probabilities. 

Assumptions  of  independence  should  be  discarded  where  they 
are  not  appropriate.  For  example,  the  binomial  probabilities 
of  failure  based  on  the  number  of  missiles  launched  should  be 
replaced  with  probabilities  that  more  accurately  reflect  the 
interaction  of  different  failure  modes,  the  knowledge  that  is 
gained  from  one  test  to  the  next,  etc.  This  would  more 
accurately  reflect  the  true  dynamics  of  the  problem.  The  use 
of  Influence  diagrams  may  make  this  a  relatively  easy  task. 

The  probabilities  associated  with  DEGRADE*,  the  part  of 
the  force  having  the  failure  mode  present,  also  needs  to  be 
refined.  The  percentages  of  degradation  and  their  likelihood 
of  occurrence  may  need  to  be  changed  to  reflect  the  concerns  of 
the  decision  maker  and  his  experts.  This  can  be  accomplished 
with  the  use  of  historical  data  and  the  judgement  of  experts  in 
the  field.  These  changes  will  result  in  different  numbers 
being  generated  by  the  analyses,  but  do  not  change  the 
structure  of  the  model. 

Model  Expansion 

The  results  of  other  than  flight  tests  can  now  be 
Incorporated  into  the  model  to  Improve  its  fidelity.  The 
results  of  aging  and  surveillance  testing  can  be  used  to  help 
predict  booster  performance.  Service  Star  data  can  be 
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Incorporated  to  help  predict  reentry  vehicle  performance.  Data 
from  simulated  electronic  launches  and  other  operational  base 
ground  tests  can  be  used  along  with  test  launch  data  to  provide 
an  overall  system  evaluation.  This  is  an  unexplored  area  but 
one  the  author  believes  may  be  of  the  most  benefit  in  advancing 
I CBM  testing.  Influence  diagrams  will  be  essential  to  help 
understand  the  complexity  of  the  relationships.  Although  this 
incorporation  of  non-flight  test  data  may  be  a  long,  iterative 
process,  full  of  discussions  and  arguments,  it  may  also  prove 
to  be  the  most  insightful. 

The  model  can  provide  an  up-to-the-minute  confidence  of 
being  able  to  detect  a  system  degrade  based  on  the  results  of 
testing  done  to  date.  For  example,  while  the  decision  maker 
may  allow  a  three  year  period  over  which  to  be  able  to  achieve 
a  level  of  confidence  in  detecting  a  degrade,  this  model  can 
easily  be  expanded  to  show  the  current  probability  of  detecting 
the  degrade  at  any  time  within  that  three  year  period. 

Differences  between  launches  can  be  taken  into  account. 

By  using  separate  variables  in  the  influence  diagrams  to 
represent  the  various  modes,  responses  to  launch  signals 
transmitted  via  ground  cables  can  be  separated  from  those 
transmitted  via  radio.  In  the  same  way,  peacetime,  fully 
capable  systems  can  be  differentiated  from  post-attack,  or 
degraded  systems. 

If  PEACEKEEPER  is  deployed  in  multiple  basing  modes, 
influence  diagrams  allow  the  respective  results  to  be 


segregated  for  individual  basing  mode  assessment,  yet  combined 
to  give  an  overall  weapon  system  analysis.  But  the  model  does 
not  have  to  be  limited  to  the  PEACEKEEPER  weapon  system.  The 
same  analyses  can  be  accomplished  for  any  system.  For  example, 
MINUTEMAN  III  missiles  based  at  F.E.  Warren  AFB  can  be  looked 
at  as  a  variable  separate  from  those  based  at  Minot  AFB  or  all 
can  be  combined  for  a  total  force  evaluation. 

Different  aspects  of  the  flight  profile  can  be  addressed. 
By  including  the  appropriate  variables,  the  probability  of 
failure(s)  occurring  in  a  particular  phase  of  flight  can  be 
analyzed.  The  effects  of  range,  reentry  angle,  type  of  reentry 
vehicle,  and  number  of  reentry  vehicles  on  accuracy  and 
survivability  can  be  looked  at  separately,  in  combination,  or 
removed  from  the  model,  depending  on  the  information  being 
sought.  Effects  of  weather  in  the  terminal  area  can  be 
included  in  the  model  and  looked  at  in  the  same  manner.  The 
degree  of  difficulty  in  doing  this  is  based  on  the  number  and 
complexity  of  the  variables  that  the  decision  maker  believes 
should  be  included.  Sensitivity  analysis  bases  in  the 
influence  diagram  should  be  of  great  help  in  identifying  the 
variables  that  have  the  most  effect  on  the  model. 

Summary 

Building  an  accurate  model  of  the  system  test  process  will 


require  extensive,  yet  relatively  simple  analyses  of  the  test 
data  that  currently  exists.  The  analyses  that  need  to  be 
accomplished  will  be  identified  through  the  building  of  the 
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Influence  diagram.  As  relationships  between  variables  are 
identified,  various  alternatives,  outcomes,  and  their 
associated  probabilities  will  be  needed  to  fill  out  the  model. 
Research  through  the  existing  historical  data  base  and 
consultations  with  experts  should  reveal  the  needed  answers. 

A  decision  analysis  model  will  not  generate  new  test  data. 
It  does  provide  a  methodology  to  show  how  the  variables  in  the 
tests  fit  together.  The  use  of  appropriate  software  tools  then 
provides  a  means  of  analyzing  both  the  data  and  the 
relationships  so  as  to  provide  the  decision  maker  with  more 
Insight  and  understanding  upon  which  to  base  his  final 
decision. 


VI I .  Conclusions 


This  pilot  decision  analysis  model,  based  on  the  decision 
analysis  methodology  and  influence  diagrams,  can  provide  the 
required  information  and  Insight  that  a  decision  maker  needs  in 
order  to  formulate  a  strategy  for  testing  the  PEACEKEEPER 
weapon  system.  The  methodology  provides  a  structure  that 
enhances  communications  between  participants  in  the  decision 
making  process  while  the  graphical  representation  provided  by 
the  use  of  influence  diagrams  is  extremely  helpful  in 
understanding  the  problem.  Building  the  influence  diagram 
requires  mapping  the  variables  under  consideration  and  their 
inter-  dependencies,  a  process  that  provides  valuable  insight 
as  the  decision  maker  is  able  to  look  at  the  problem  as  a  whole 
Instead  of  as  a  series  of  unconnected  events. 

Assumptions  are  made  explicit  in  the  model  and  can  be 
easily  seen  and  challenged.  Influence  diagrams,  because  of 
their  flexibility,  allow  many  assumptions  to  be  relaxed. 

Finally,  the  model  can  be  effectively  analyzed  on  a 
microcomputer  using  the  Performa  decision  analysis  software 
program.  Analysis  of  the  pilot  model  has  shown  the  ability  to 
provide  information  concerning  the  maximum  expected  value  of 
the  complete  model  and  for  each  level  of  degradation. 
Probabilities  of  a  degrade  being  present  in  the  force,  seeing 
the  occurrence  of  at  least  one  failure,  and  identifying  the 
failure  mode  were  analyzed. 


Data  used  for  this  thesis  can  be  easily  replaced  with 
actual  data.  By  this  simple  step,  the  model  can  provide  more 
useful  information  to  a  decision  maker  than  was  here-to-fore 
possible . 
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